Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable traditional threading as an option #3149

Open
wants to merge 5 commits into
base: develop
Choose a base branch
from
Open

Enable traditional threading as an option #3149

wants to merge 5 commits into from
+95 −33

Conversation

aerorahul
Copy link
Contributor

@aerorahul aerorahul commented Dec 10, 2024 •
edited
Loading

Description

In a previous PR, stubs were added to enable traditional threading in the UFS.
This PR takes the next step to execute the model with traditional threading.
Specifically;

  1. Do not use the _esmf ufs_configure file for traditional threading. This sets the globalResourceControl to false (traditional threading) ... this was done in the previous PR
  2. when calculating components pet bounds do not multiply with number of threads also in ufs_configure ... this PR
  3. export OMP_NUM_THREADS=$UFS_THREADS ... this was done in the previous PR

The default behaviour to use ESMF-managed threading is retained. Traditional threading can be achieved by toggling USE_ESMF_THREADING flag in config.fcst

Closes #3122 - as in allows the model to be configured to run w/ traditional threading. Testing on C768 S2SW was successful on Hercules (This was previously reported to be failing)

The use of traditional threading will be inefficient for model components that are not thread-enabled and supported e.g. MOM6 and CICE6, but since they require smaller number of tasks compared to ATM or WAV, the CPU 'waste' is not huge.

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
    • EMC verif-global
    • GDAS
    • GFS-utils
    • GSI
    • GSI-monitor
    • GSI-utils
    • UFS-utils
    • UFS-weather-model
    • wxflow

How has this been tested?

Testing on Hercules for a C768 S2SW case was run with traditional threading (4 threads)
Some particulars of the run are as follows:
Run directory: /work/noaa/stmp/rmahajan/HERCULES/RUNDIRS/c768s2swtt/gfs.2019120300/gfsfcst.2019120300/fcst.2077795/
Run log: /work2/noaa/stmp/rmahajan/RUNTESTS//COMROOT/c768s2swtt/logs/2019120300/gfs_fcst_seg0.log

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

@aerorahul aerorahul marked this pull request as ready for review December 10, 2024 22:24
@aerorahul
Copy link
Contributor Author

After some waiting for WW3 points, it seems the model progressed in the test on Hercules. The run will time out at 30 mins, because I was testing it. Still worth taking a look at the log.

@aerorahul
Copy link
Contributor Author

A run of the test C768_S2SW succeeded on Hercules with traditional threading.
This test was previously reported to be failing on Hercules and Orion.
I have not tested this for C1152 or Orion.

@aerorahul aerorahul mentioned this pull request Dec 11, 2024
Open
@aerorahul aerorahul requested a review from RuiyuSun December 11, 2024 14:51
@WalterKolczynski-NOAA WalterKolczynski-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Dec 11, 2024
@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Dec 11, 2024
@aerorahul
Copy link
Contributor Author

@JessicaMeixner-NOAA is there an expectation the model reproduces results based on who is managing threading?

@JessicaMeixner-NOAA
Copy link
Contributor

I would think that we should get the same answers regardless of if it's ESMF managed threading or not, or how many threads. I don't know if ufs-weather-model currently tests, but assume that was probably an early test if not a current one.

@emcbot emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Dec 11, 2024
@emcbot
Copy link

emcbot commented Dec 11, 2024

CI Passed on Hera in Build# 1
Built and ran in directory /scratch1/NCEPDEV/global/CI/3149


Experiment C48_ATM_26a0c420 Completed 2 Cycles: *SUCCESS* at Wed Dec 11 19:51:36 UTC 2024
Experiment C48mx500_3DVarAOWCDA_26a0c420 Completed 2 Cycles: *SUCCESS* at Wed Dec 11 20:09:53 UTC 2024
Experiment C48mx500_hybAOWCDA_26a0c420 Completed 2 Cycles: *SUCCESS* at Wed Dec 11 20:22:02 UTC 2024
Experiment C96_S2SWA_gefs_replay_ics_26a0c420 Completed 1 Cycles: *SUCCESS* at Wed Dec 11 20:28:51 UTC 2024
Experiment C96_atm3DVar_26a0c420 Completed 3 Cycles: *SUCCESS* at Wed Dec 11 21:35:31 UTC 2024
Experiment C96C48_hybatmDA_26a0c420 Completed 3 Cycles: *SUCCESS* at Wed Dec 11 21:35:31 UTC 2024
Experiment C96C48_hybatmaerosnowDA_26a0c420 Completed 3 Cycles: *SUCCESS* at Wed Dec 11 21:35:31 UTC 2024
Experiment C96C48_ufs_hybatmDA_26a0c420 Completed 3 Cycles: *SUCCESS* at Wed Dec 11 22:12:05 UTC 2024
Experiment C48_S2SW_26a0c420 Completed 2 Cycles: *SUCCESS* at Wed Dec 11 22:12:30 UTC 2024
Experiment C48_S2SWA_gefs_26a0c420 Completed 1 Cycles: *SUCCESS* at Wed Dec 11 23:03:30 UTC 2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DusanJovic-NOAA DusanJovic-NOAA DusanJovic-NOAA approved these changes

@RuiyuSun RuiyuSun RuiyuSun approved these changes

@WalterKolczynski-NOAA WalterKolczynski-NOAA WalterKolczynski-NOAA approved these changes

@JessicaMeixner-NOAA JessicaMeixner-NOAA JessicaMeixner-NOAA approved these changes

@junwang-noaa junwang-noaa Awaiting requested review from junwang-noaa junwang-noaa is a code owner

Assignees
No one assigned
Labels
CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Traditional threading as option for forecast model
6 participants
@aerorahul @JessicaMeixner-NOAA @emcbot @WalterKolczynski-NOAA @RuiyuSun @DusanJovic-NOAA